Dietary assessment system and method

ABSTRACT

The present system and method provides a more precise way to record food and beverage intake than traditional methods. The present disclosure provides custom software for use in mobile computing devices that include a digital camera. Photos captured by mobile digital devices are analyzed with image processing and comparisons to certain databases to allow a user to discretely record foods eaten. Specifically, the user captures images of the meal or snack before and after eating. The foods pictured are identified. Image processing software may identify the food or provide choices for the user. Once a food is identified and volume of the food is estimated, nutrient databases are used for calculating final portion sizes and nutrient totals.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 61/191,048, filed Sep. 5, 2008, the entire contents of which ishereby incorporated by reference.

STATEMENT REGARDING GOVERNMENT FUNDING

This invention was made with government support under U01 CA130784 andR01 DK073711 awarded by the National Institutes of Health. Thegovernment has certain rights in the invention.

BACKGROUND

1. Technical Field

This application relates to the fields of mobile phones and computingdevices, digital photographs, image processing, and food databases. Inparticular, this application relates to the use of the above-mentionedfields in a system for recording dietary intake and analyzingnutritional content.

2. Description of the Related Art

Dietary intake provides some of the most valuable insights for mountingintervention programs for prevention of chronic diseases like obesity,for example. However, accurate assessment of dietary intake isproblematic. Emerging technology in mobile telephones (cell phones) withhigher resolution images, improved memory capacity, and fasterprocessors, allow these devices to process information not previouslypossible. Mobile telephones and PDAs (personal digital assistants),which are widely used throughout the world, can provide a uniquemechanism for collecting dietary information that reduces the burden onrecord keepers. Indeed, a dietary assessment application on a mobiletelephone would be of value to practicing dietitians and researchers.

The increasing prevalence of obesity among younger generations is ofgreat concern and has been linked to an increase in type 2 diabetesmellitus. Accurate methods and tools to assess food and nutrient intakeare essential in monitoring the nutritional status of this age group forepidemiological and clinical research on the association between dietand health. The collection of food intake and dietary informationprovides some of the most valuable insights into the occurrence ofdisease and subsequent approaches for mounting intervention programs forprevention.

With this growing concern for obesity, the need to accurately measurediet becomes imperative. Assessment among adolescents is problematic asthis group has irregular eating patterns and has less enthusiasm forrecording food intake. Early adolescents, ages 11 to 14 years, inparticular, are in that period of time when the novelty and curiosity ofassisting in or self-reporting of food intakes starts to wane and theassistance from parents is seen as an intrusion. Dietary assessmentmethods need to continue to evolve to meet these challenges. There isrecognition that further improvements will enhance the consistency andstrength of the association of diet with disease risk, especially inlight of the current obesity epidemic among this group.

Preliminary studies among adolescents suggest that innovative use oftechnology may improve the accuracy of diet information from youngpeople. PDAs are ideal as a field data collection device for dietassessment; however, there have been problems when deploying these typesof devices if one does not understand the user and the environment inwhich the device be deployed. Minimal training using mobile devices mayimprove the accuracy of recording.

The assessment of food intake in adolescents has previously beenevaluated using more traditional methods of recording, i.e., food record(FR), the 24-hour dietary recall (24HR), and a food frequencyquestionnaire (FFQ), with external validation by doubly-labeled water(DLW) and urinary nitrogen. Currently, there are too few validationstudies in children to justify one particular method over another forany given study design.

A review of some of the most popular dietary assessment methods isprovided below. This review demonstrates the significance of the mobilesystem of the present disclosure which can be used for population andclinical based studies to improve the understanding of dietary exposuresamong adolescents.

24-Hour Dietary Recall

The 24-hour dietary recall (24HR) consists of a listing of foods andbeverages consumed the previous day or the 24 hours prior to the recallinterview. Foods and amounts are recalled from memory with the aid of aninterviewer who has been trained in methods for soliciting dietaryinformation. A brief activity history may be incorporated into theinterview to facilitate probing for foods and beverages consumed. TheFood Surveys Research Group (FSRG) of the United States Department ofAgriculture (USDA) has devoted considerable effort to improving theaccuracy of this method.

The major drawback of the 24HR is the issue of underreporting of thefood consumed. Factors such as obesity, gender, social desirability,restrained eating and hunger, education, literacy, perceived healthstatus, age, and race/ethnicity have been shown to be related tounderreporting. For example, significant underreporting of large foodportions is found when food models showing recommended serving sizeswere used as visual aids for respondents. Given that larger foodportions have been observed as occurring over the past 20 to 30 yearsthis may be a contributor to underreporting and methods to captureaccurate portion sizes are needed. In addition, youth, in particular,are limited in their abilities to estimate portion sizes accurately. Themost common method of evaluating the accuracy of the 24HR with childrenis through observation of school lunch and/or school breakfast andcomparing foods recalled with foods either observed as eaten or foodsactually weighed. These recalls have demonstrated both under-reportingand over-reporting, and incorrect identification of foods.

The Food Record

The 24HR is useful in population based studies; however, the preferreddietary assessment method for clinical studies is the food record. Forthe food record, participants are asked to record all food and beveragesconsumed throughout a 24-hour period. Depending on the primary nutrientor nutrients, or foods of interest, the minimum number of food recordsneeded is rarely less than two days. Training the subjects, telephoningwith reminders for recording, reviewing the records for discrepancies,and entering the dietary information into a nutrient database can take alarge amount of time and requires trained individuals.

The food record is especially vulnerable to underreporting due to thecomplexity of recording food. It has been shown that 10-12 year oldchildren significantly underreport total energy intake (TEI) when theintake is compared against an external marker, doubly-labeled water. Inaddition, as adolescents snack frequently, have unstructured eatingpatterns, and consume greater amounts of food away from the home, theirburden of recording is much greater compared to adults. It has beensuggested that these factors, along with a combination of forgetfulnessand irritation and boredom caused by having to record intake frequentlymay be contributing to the underreporting in this age group. Dietaryassessment methods perceived as less burdensome and time consuming mayimprove compliance.

Portion Size Estimation

Portion size estimation may be one contributor to underreporting, ingeneral. For example, it has been found that training in portion sizeestimation among 9-10 year olds significantly improves estimates forsolid foods measured by dimensions or cups, and liquids estimated bycups. Amorphous foods are estimated least accurately even after trainingand other foods still present significant errors.

Evaluation of Dietary Assessment Methods

The number of days needed to estimate a particular nutrient depends onthe variability of the nutrient being assessed and the degree ofaccuracy desired for the research question. Most nutrients require morethan four days for a reliable estimate. However, most individuals wearyof keeping records beyond four days which may decrease the quality ofthe records.

Another challenge in evaluating dietary assessment methods is comparingthe results of the dietary assessment method to some measure of “truth.”This is best achieved by identifying a biomarker of a nutrient ordietary factor. The underlying assumption of a biomarker is that itresponds to intake in a dose-dependent relationship. The two methodsthat have widest consensus as valid biomarkers are DLW for energy and24-hour urinary nitrogen for protein intake. A biomarker does not relyon a self-report of food intake, thus theoretically the measurementerrors of the biomarker are not likely to be correlated with those ofthe dietary assessment method. Other biomarkers collected from urinesamples may include potassium and sodium. Some plasma or serumbiomarkers that have been explored are levels of ascorbic acid forvitamin C intake, and 13-carotene for fruits and vegetables orantioxidants. These latter markers are widely influenced by factors suchas smoking status and supplement use, thus their interpretation toabsolute intake is limited.

SUMMARY OF THE DISCLOSURE

The present system and method provides a mobile device (e.g., a PDA ormobile telephone) food record that provides an accurate account of dailyfood and nutrient intake among adolescents or others. The present systemand method uses a mobile network connected device that has a camera totake images of food before and after it is consumed and estimate foodintake using image analysis methods. Images of food can also be markedwith a variety of input methods that link the item for image processingand analysis to estimate the amount of food. Images acquired before andafter foods are eaten can estimate the amount of food consumed. Providedare alternatives for recording food consumed when images cannot beobtained for the mobile food record.

Provided is a cell phone, PDA, digital camera, or other device tocapture both visual and recorded detail that is electronically submittedto a researcher, thereby easing respondent and researcher burden, andproviding accurate estimates of nutrient, food, and supplement intakes.Provided in the present disclosure is imaging software for use withdigital images that estimates quantities of foods consumed, modificationof the FNDDS nutrient database, and a user-friendly interface.

The present disclosure includes the use of image analysis tools foridentification and quantification of food consumption. Provided to aidin image analysis are fiducial markers of known sizes to be included indigital photographs taken by the user. Images obtained before and afterfood is consumed are used to estimate the diet of an individual,including energy and nutrient content. Also provided are applications ofthe mobile food record for helping users make healthier eating decisionsbased on analyzed images taken before eating.

Other advantages and features will be apparent from the followingdetailed description when read in conjunction with the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the disclosed methods andapparatuses, reference should be made to the embodiments illustrated orpictured in greater detail in the accompanying drawings, wherein:

FIGS. 1( a) through 1(d) represent examples of input techniques for usewith mobile device food record. Specifically, FIG. 1( a) illustrates atree method to select a food item from the database; FIG. 1( b)illustrates a search method to select a food item from the database;FIG. 1( c) illustrates a tree method to mark digital pictures with afood item in the database; and FIG. 1( d) illustrates a search method tomark digital pictures with a food item in the database.

FIGS. 2( a) through 2(d) represent additional input devices and labelsfor use with mobile device food record. Specifically, FIG. 2( a)illustrates the use of a stylus to hand write notes to label food itemsin a digital picture of a meal; FIG. 2( b) illustrates the use of anon-screen tree method to label food items in a digital picture of ameal; FIG. 2( c) illustrates the use of an on-screen keyboard to searchfor a food item from a database; and FIG. 2( d) illustrates a labeledmeal.

FIG. 3 is an image of a meal taken on a tablecloth having a pattern ofsquares of known size to permit calibration of the image.

FIG. 4 is a diagram illustrating an exemplary embodiment of an imageprocessing and analysis system of the present disclosure.

FIGS. 5( a) and 5(b) show images of segmented food items, with FIG. 5(a) including a food item segmented using a fix threshold, and FIG. 5( b)including additional food item segmentation using color information.

FIGS. 6( a) and 6(b) show images of classified food items, wherein inFIG. 6( a), all food items are successfully classified using a SVM, andwherein in FIG. 6( b), some food items are misclassified by the SVM.

FIG. 7 is a diagram illustrating an exemplary embodiment of the mobiletelephone food record of the present disclosure.

It should be understood that the drawings are not necessarily to scaleand that the disclosed embodiments are sometimes illustrateddiagrammatically and in partial views. In certain instances, detailswhich are not necessary for an understanding of the disclosed methodsand apparatuses or which render other details difficult to perceive mayhave been omitted. It should be understood, of course, that thisdisclosure is not limited to the particular embodiments illustrated orpictured herein.

DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EMBODIMENTS

An embodiment of the present disclosure uses a network-connected mobiledevice that has a built-in camera to take images of food before andafter it is consumed and estimate food intake using image analysismethods. Mobile devices, such as PDAs and mobile telephones withcameras, are general purpose computing devices that have a great deal ofcomputational power that can be exploited for this purpose.

The present system and method provides a mobile device that isattractive to users and which can be used to measure food intake. Mostmobile devices include digital cameras which make taking pictures andlabeling the content of the pictures less burdensome than writing onpaper. The use of a mobile device that works the way young peopleinteract with portable devices may address many of the issues outlinedas barriers to recording food intake among adolescents. Young userstreat their mobile device as an extension of their personality and thishas been considered in the design of our system.

Mobile devices have the potential to create an entire new platform forapplications and services that could be used for dietary assessments.For example, some individuals may forget to record their food wheneaten, in which case the record can become a cross between a recall anda record. With paper and pencil recording, there is no way a researchercan check that foods were recorded at the time of the meal or that allmeals were recorded at the end of the day. With a mobile device, everyentry records a time stamp, thus allowing researchers to more accuratelydetermine if data entry occurred at typical meal times (record), longafter the meal, or all at once at the end of the day (an unassistedrecall). The use of an image provides another dimension of verifyingfood intake.

In one embodiment of the present disclosure uses a mobile device with abuilt-in camera, integrate image analysis, and visualization tools witha nutrient database, to allow an adolescent user to discretely “record”foods eaten. The user captures an image of his/her meal or snack bothbefore and after eating. The present disclosure provides automatic imageanalysis methods to determine the food consumed, thereby reducing theburden of many aspects of recording for the users and reduces analysisfor the researchers.

In one embodiment, software in the mobile device prompts the user to“record a new meal,” “review meals,” or “alternate method.” Thereafter,should the user choose “record a new meal,” the user may be presentedwith prompts for breakfast, lunch dinner, or snack. The software maythen guide the user to capture images of the food and beverage beforeand after eating, including various reminders (inclusion of fiducialmarkers, for example). Before and after images may be providedside-by-side to the user before exiting the program. Alternatively, thesoftware may give the user the option of “I ate everything” rather thancapturing an image of completely empty plates and glasses. In anotherembodiment, the user may choose the option “review meals” to reviewpreviously recorded images. This may be especially useful when images ofa meal were recorded, but identification and tagging of the food has notyet occurred. In this embodiment, viewing and labeling the saved imagescan proceed with or without the searching databases, and based on thelabels assigned, energy and nutrient content of food consumed can bedetermined.

However, in the event that a picture of a food cannot be obtained, dueto technical difficulties or otherwise, the system includes analternative way of determining the food consumed. In one illustratedembodiment of the present disclosure, the user may identify the foodsconsumed by writing on the screen, by tapping the screen, or by usingvarious data entry menus and forms. Entry methods include selectingfoods from a tree list, searching for a food in the database using wordsor portions of words, or other suitable data entry techniques. Examplesof these input techniques are shown in FIG. 1 and FIG. 2.

FIG. 1( a) illustrates a mobile device in which a user can use a menutree method to select a food item from a database. FIG. 1( b)illustrates using a search term such as “muffin” to select a food itemfrom the database. FIG. 1( c) illustrates use of a tree method to markdigital pictures with the food item from the database. After the pictureis taken, the user can pull up the menu and mark the picture food itemfrom the database. FIG. 1( d) illustrates use of the search method tomark digital pictures of a food item. After a picture of the food itemis taken using the mobile device, the user can then search for a termsuch as “corn bread muffin” and then mark the photo with the appropriatefood from the database.

In addition to the described methods of recording, audio entry of thefoods consumed along with their portion size is also contemplated.Recording in this manner requires voice recognition software, but may beuseful in situations where technical difficulties or forgetfulnessprevents the user from taking a picture before eating.

Additional input devices and labels for use with a mobile device foodrecord are illustrated in FIGS. 2( a) through 2(d). FIG. 2( a)illustrates use of a stylus to hand write notes and label food items ina digital picture of the meal to help with proper identification orclassification of the food items that were eaten. FIG. 2( b) illustratesuse of an on-screen tree method to label food items in a digitalpicture. FIG. 2( c) illustrates use of an on-screen keyboard to searchfor a food item in a database or to type in information related to theitem in the photograph. FIG. 2( d) illustrates a labeled meal using thesystem of the present invention (not shown are the secondary prompts).Once certain labels are added such as “salad”, secondary prompts may beused to ask the user regarding additional elements such as the type ofsalad dressing, the amount of butter or oil on the bread, any cheeseadded or the like. These additional prompts help the user capture addedquantities of food which may not be detectable in the image of the meal.

Alternatively, a computing device having a database, or access to adatabase of food image parameters and software for analyzing andcomparing the same with pictures taken may itself be able to limit thenumber of choices of what a food might be. For instance, the computingdevice may be capable of determining that a tan piece of meat is pork,chicken, or fish, or that a food item is a green vegetable. In thisembodiment, the user would confirm the suggestions rather than startingan entry from “square one.”

In addition, the device may prompt users with real-time reminders sentusing the network capabilities of the mobile device. Reports can be sentto a central system to allow regular monitoring. The use of time-stampedfood entries and images can aid the research dietitian or the clinicaldietitian in reviewing the food record with or without the adolescent toidentify foods.

Image Analysis and Visualization

An illustrated embodiment of the present disclosure includes a method toautomatically estimate the food consumed at a meal from an imageacquired from a mobile device. An example of such an image is shown inFIG. 3. The present system identifies each food item and estimates thevolume of each food item consumed. From this information, the energy andnutrients consumed can be determined. This is not an easy problem inthat some foods may not be identifiable from an image. For example, thetype of milk in a cup (e.g., low fat or skim milk) may not be determinedfrom the image alone. This requires other types of “side information” beavailable to the system either through how the food was packaged (e.g.,an image of the milk carton) or through inputs (manual or audio) fromthe user.

An illustrated embodiment of the present invention includes acalibrated, possibly 3D, imaging system. A block diagram of an exemplaryimage analysis system is shown in FIG. 4. The present system and methodfor addressing the various tasks is described below.

Image Calibration and Acquisition

An illustrated embodiment of the present disclosure includes a 3Dcalibrated image system to facilitate determining how much food wasconsumed. In one embodiment, the user takes the image with a knownfiducial object placed next to the food to “calibrate sizes” in theimages. A pen or PDA stylus may be used as a fiducial in an illustratedembodiment. Known dimensions of a plate or cup in a scene may be used tocalibrate the image. Other information in the scene such as the patternon the tablecloth (see FIG. 3) could also be used. The squares on thetablecloth have a known size for calibration.

For 3D or volume estimation, multiple images of the scene taken atdifferent orientations may also be used. This also requires thatcalibration information be available so that depth information can berecovered.

Image Segmentation

In an illustrated embodiment of the present disclosure, the systemautomatically determines the regions in the image where a particularfood is located. This can be accomplished using a combination of edgedetection and color analysis, for example. Once a food item issegmented, the system identifies the food item and then estimates howmuch food is present in the image.

An illustrated image segmentation method is a two step process. In thefirst step the image is converted to a grayscale image and thenthresholded with a threshold of 127 to form a binary image. It wasdetermined empirically that the pixel values in the binary imagecorresponding to a plate had values of 255. For segmenting the fooditems on the plate, the binary image was searched in 8-point connectedneighbors for the pixel value 0. Connected segments less than 1000pixels were ignored because they correspond to the tablecloth (see FIG.3). In this step a fixed threshold was illustratively used. Thus, pixelscorresponding to the food items might be considered as the plate. As aresult, a second step of segmentation is included in an illustratedembodiment. The result of the first step of segmentation is shown inFIG. 5( a).

In the second step, the image is first converted to the YCbCr colorspace. Using the chrominance components, Cb and Cr, the mean value ofthe histogram corresponding to the plate was found. Pixel locationswhich were not segmented during the first step are then compared withthe mean value of the color space histogram of the plate to identifypotential food items. These pixels were labeled differently from that ofthe plate. Then 8-point connected neighbors for the labeled pixels weresearched to segment the food items. An example is shown in FIG. 5( b),the food item, i.e. scrambled egg, which was not segmented in the firststep is successfully segmented in the second step using the color space.

Feature Extraction

Two types of features may be extracted from each segmented food region,namely color features and texture features. For color features, theaverage value of the pixel intensity along with the two color componentsis extracted. For texture features, Gabor filters to measure localtexture properties in the frequency domain may be used.

Gabor filters describe properties related to the local power spectrum ofa signal and have been used for texture features. A Gabor impulseresponse in the spatial domain consists of a sinusoidal plane wave ofsome orientation and frequency, modulated by a two-dimensional Gaussianenvelope and is given by:

${h\left( {x,y} \right)} = {{\exp\left\lbrack {{- \frac{1}{2}}\left( {\frac{x^{2}}{\sigma_{x}^{2}} + \frac{y^{2}}{\sigma_{y}^{2}}} \right)} \right\rbrack}{\cos\left( {{2\;\pi\;{Ux}} + \varphi} \right)}}$

In one illustrated embodiment of the present disclosure, a Gaborfilter-bank was used. It is highly suitable for our use where thetexture features are obtained by subjecting each image (or in our caseeach block) to a Gabor filtering operation in a window around each pixeland then estimate the mean and the standard deviation of the energy ofthe filtered image. A Gabor filter-bank consists of Gabor filters withGaussians of several sizes modulated by sinusoidal plane waves ofdifferent orientations from the same Gabor-root filter as defined in theabove equation, it can be represented as:g _(m,n)(x, y)=a ^(−m) h({tilde over (x)}, {tilde over (y)}), a>1where {circumflex over (x)}=a^(−m)(x cos θ+y sin θ), ŷ=a^(−m)(−x sin θ+ycos θ), θ=nx/K (K=total orientation, n=0, 1, . . . , K−1, and m=0, 1, .. . , S−1), and h(-,-) is defined in Equation (1). Given an imageI_(g)(r,c) of size H×W, the discrete Gabor filtered output is given by a2D convolution:

${{I_{g_{m,n}}\left( {r,c} \right)} = {\sum\limits_{s,t}{{{I_{E}\left( {{r - s},{c - t}} \right)}g_{m,n}} \star \left( {s,t} \right)}}},$

As a result of this convolution, the energy of the filtered image isobtained and then the mean and standard deviation are estimated and usedas features. In our implementation, we divided each segmented food itemin to 64×64 non-overlapped blocks and used Gabor filters on each block.We used the following parameters: 4 scales (S=4), and 6 orientations(K=6).

Classification

Once the food items are segmented and their features are extracted, thenext step is to identify the food items using statistical patternrecognition techniques. For classification of the food item, a supportvector machine (SVM) may be used. The feature vectors presently used forthe SVM contain 51 values, 48 texture features and 3 color features. Thefeature vectors for the training images (which contain only one fooditem in the image) were extracted and a training model was generatedusing the SVM. A set of 17 images were used as training images and eachfood item was given a unique label.

An illustrated embodiment of the present disclosure includes a databaseof images taken using a digital camera and plastic food replicas in alaboratory. The images were acquired using specific conditions, suchthat the foods were placed on a white plate on a checkerboard (black andwhite) patterned tablecloth. The tablecloth was used as a fiducial markfor estimating the dimensions and area of the food item. The whiteplates were used to assist the segmentation of the food items. Thetraining images used were taken with only one food item and the testingimage contained 2 or 3 food items. The database consists of 50 images,17 images were used for training and 33 images were used for testing.The average classification results indicated a 93.745% accuracy ratewhen 17 images were used as training images and 14 images containing 32food items were used as test images. FIGS. 6( a) and 6(b) show images ofclassified food items, wherein in FIG. 6( a), all food items aresuccessfully classified using a SVM, and wherein in FIG. 6( b), somefood items are misclassified by the SVM, i.e., beef roast ismisclassified as steak.

Volume Estimation

Based on the segmentation and reference size estimation, the presentsystem and method determines the volume of food consumed in cm³. Formany foods this may not be possible from one image of the meal. Thesystem and method may use multiple images and computer visualizationmethods using 3D shape reconstruction techniques to improvedetermination of the volume of food consumed. The present system andmethod generates a similar shaped object as the food item as a referenceand composite it over the food item in the image and ask the user toadjust the shape to be smaller or larger to increase our accuracy involume estimation. This involves a combination of image processing andgraphical rendering techniques to correctly position and size theestimated volume shape. The user may simply adjust a slider bar or useanother suitable input to confirm the choice of the size. Thisinteractive user adjustment is an option in the final deployed system asthe estimation algorithms increase in accuracy.

Estimating Food Consumed

By using digital images taken before and after the meal is eaten, thepresent system and method estimates the amount of food consumed. Imagesubtraction may be used to determine the total volume of each foodobject consumed. Once the volume in cm³ of each food item is determined,this information is combined with the portion code and portion weight inthe USDA Food and Nutrient Database for Dietary Studies (FNDDS) todetermine the gram weight of the food. For example, for homemadechocolate pudding, the food code is 13210220 in FNDDS. The portion code,10205, associated with the chocolate pudding is 1 cup. The portionweight is 261 grams and 1 US cup is known to be 236.588237 cm³. If theportion is calculated to be 100 cm³, then the gram weight of the portioncode and the cm³ of the portion code are used to solve for the gramweight of the portion size. In this case the gram weight would be110.318248 grams. Once the gram weight is determined the nutritionalinformation in the FNDDS for each food item can be used to determine thefinal energy and nutritional content of the consumed food. The FNDDSdoes not have built in volume measures so each portion code and portionweight needs to have an additional cubic centimeter field added toaddress the conversion.

In the FNDDS, there are food items that have volume measures that areaccompanied with a gram weight and these volume measures are: cubicinch, cup, tablespoon, teaspoon, and fluid ounce. For the remaining fooditems, methods need to be developed to efficiently, yet accuratelyprovide the necessary volumetric measurement to achieve the conversionto gram weight. The number of foods with adequate information and thenumber of foods with inadequate information are shown in Table 1.

TABLE 1 Frequency of food items (main items and additional items) byvolumetric measure in FNDDS Volumetric measure Number of foods with inFNDDS the volumetric measure Cubic inch 2,002 Fluid ounce 744 Cup 7,388Tablespoon 686 Teaspoon 75 None of the above 2,925 TOTAL 13,820

Food items with no volumetric measure are commonly eaten foods, such asbagels, doughnuts, pizza, turnovers, rolls, sandwiches, pie, granolabars, some cookies, muffins, ice cream cones, bread, candy bars. Thevolumetric measurements for these items include such portion codedescriptions as 1 miniature, 1 package, 1 sandwich, 1 loaf, 1 roll, 1medium, and 1 small. There are also some candies that have a portioncode description of 1 piece. As another example, there are 94 items witha portion code description of cupcake. The numbers in Table 1 do notinclude the subcode foods, which represent many candies and somecommercial snack cakes.

Suggested approaches to determining volume include volume measurementsin FNDDS. In some cases, a food item may have more than one of thesevolume measures associated with it. The number of foods (main food itemsonly) with multiple volumetric measurements is shown in Table 2. InFNDDS, additional food items have the same code as a main food item andsubcode food items are linked to a main food item, this table onlyincludes the total main food items (6974 items). For these foods, adecision tree can be developed as to which volume measure wouldpreferably be used for conversion calculations. Since cubic inch issimilar to cubic centimeter, this would be a natural first choicemeasure when a food contains cubic centimeter as part of its portioncode description in the FNDDS. A beverage, such as milk, would haveportion code descriptions of cup and fluid ounces. Thus, a decisionwould need to be made as to a preference for cup or fluid ounces.Proposed is a ranking of volumetric measures by preference to be 1)cubic inch, 2) fluid ounce, 3) cup, 4) tablespoon, and 5) teaspoon. Anyfood item with at least one of these measures in the portion codedescription contains the elements needed to convert total volumeconsumed to gram weight.

As evident in this Table 2 below, 1,369 main food items do not have anyvolumetric measure. This accounts for about 20% of the main food codeslisted in FNDDS. Those with multiple volumetric measures are listed inthe order of suggested preference stated above. For example, those fooditems with both cubic inch and cup would preferentially defer to thecubic inch volumetric measurement. And, those with any other volumetricmeasurement and cubic inch would also defer to cubic inch. Then, fluidounce would be the volumetric measurement of preference, and so on.

TABLE 2 Frequency of food items (main items) with more than onevolumetric measure in FNDDS Cubic Fluid Table- Tea- Inch Ounce Cup spoonspoon None Total Cubic Inch 245 2 779 3 Fluid Ounce 269 6 7 Cup 389 3398284 4 Tablespoon 192 4 Teaspoon 23 None 1369 Total 6974

Physical measurements of foods without volume measurements inFNDDS—Those food items that do not have a volumetric measurement in theportion code may have size measurements in the portion code. The volumefor these items could be calculated. An example would be: 1-layer cake(8″ or 9″ diameter, 1½″ high). Other food items may have part of themeasurement in the portion code, and thus would require additionalmeasurements in order to calculate volume. For example, a 2″×1″ piece ofcake without the height measurement given or a large apple with a 3″diameter. For these food items, samples of food items could be purchasedand measured. Then weight of sample purchased (example a large applewith a 3″ diameter) would be compared with weight listed in FNDDS. Itemswould be measured with a caliper and at least three measurements wouldneed to be made for purposes of accuracy.

Measurement of volume of foods without volume measurements in FNDDS—Forfood items which have no volumetric measurement or size measurements inthe portion code, volume could be determined using other methods, suchas liquid, gas, or solid displacement. For example, volume measurementscould be done using the method of liquid displacement for either foodswrapped in saran and sealed or Nasco® plastic foods. The plastic foodsare made to reproduce actual food in commonly eaten portion sizes. Thecompany provides the weight, size, and/or portion size of these fooditems. Additionally, volumes of foods can be determined using rapeseed(solid) displacement. These displacement methods require skilledtechnique, especially the solid displacement method. It should also benoted that one of the challenges presented with varying forms ofmeasuring foods is that there is not consistency and thus accuracy iscompromised.

Computation of density of foods using formula for true density—Densityis defined as the amount of mass of a material per unit volume. Particleand bulk density are used to define the mass per volume of particlesalone or per volume of a group of particles. Density can be calculatedby using the make-up of the major food components of water,carbohydrate, protein, fat, ash, and ice and by using a formula. (See,Physical Properties of Foods by Serpil Sahin and Servet Gulum Sumnu,page 22-24, and Handbook of Food Engineering, Second Edition by DennisR. Heldman and Daryl B. Lund CRC Press Publisher 2007, pages 399-403,which are both incorporated by reference herein). Densities of all fooditems in the FNDDS could be calculated using this formula, thusproviding consistency. However, it would be prudent to test how wellthis calculated true density correlates with the density computed fromfood items in FNDDS which have adequate information (weight and volume).

There is not a value for ash in the nutrient values, thus mineralcomponents would have to be added to determine the value for ashcontent. The density calculated using this formula is a true density andis a density of a pure substance or a composite material calculated fromthe densities of its components considering conservation of mass andvolume. The formula for true density uses: 1) density of the foodcomponent, (kg/m³) which is dependent on temperature, 2) volume fractionof the food component, and 3) mass fraction of the food component. Thereare specific formulas for the density of each of the food componentsusing the temperature (T) of the food. The formula for density (p) ofthe food item at a particular temperature is as follows:

${\rho\mspace{14mu}\left( {{kg}\text{/}m^{3}} \right)} = \frac{1}{\Sigma\left( {X_{i}/\rho_{i}} \right)}$where X is the mass fraction of the component of the food and ρ is thedensity of that component at a particular temperature T. The formulasfor density of the food components at a particular temperature are asfollows:True density of water=997.18+3.1439×10⁻³ T−3.7574×10⁻³ T ²True density of carbohydrate=1599.1−0.31046TTrue density of protein=1330−0.1584TTrue density of fat=925.59−0.41757TTrue density of ash=2423.8−0.28063TTrue density of ice=916.89−0.1307TWhere true densities are in kg/m³ and temperatures (T) are in ° C. andvaries between −40 and 150° C.

FIG. 7 summarizes some steps of the mobile telephone food record systemof the present disclosure in a diagram format. In step 1, the useracquires an image of the food before and after eating with the mobiledevice. A digital image is different from a photograph in that usefulinformation, called metadata is captured that is not visible, such asthe time stamp and digital codes. Thereafter, the image, along withmetadata, is sent to a server. In steps 2 and 3, image analysis occursat the server. The food items are identified (labeled and the volume ofeach food estimated). The results are then sent back to the user in step4. In step 5, the user confirms and/or adjusts the information and sendsit back to the server. The data is indexed with a nutrient database, theFNDDS, to compute the energy and nutrient content of the foods consumedin step 6. And finally, in step 7, the results can be sent toresearchers or dietitians.

Alternatively, given the capacity of mobile devices to store largeamounts of information and software, computation of volume and nutrientvalues may proceed without a server. Instead, all computations wouldoccur in the mobile device alone, the user being capable of confirmingand/or adjusting the results, as described above. In either case, inanother application of the mobile device recording system, after takinga picture of the food, and before eating, the user can be directlyprovided with an estimate of the nutritional information for the servedportion. Further, this information could be combined with a computationof the total nutrient and energy intake of that day, for example. Inthis application of the mobile food record, the user is provided withinformation that may help in making healthier food choices and possiblymaintain or loose weight.

While only certain embodiments have been set forth, alternatives andmodifications will be apparent from the above description to thoseskilled in the art. These and other alternatives are consideredequivalents and within the spirit and scope of this disclosure and theappended claims.

1. A method of measuring food intake of a user, the method comprising:providing a mobile device having a built-in camera; capturing a firstimage of food to be eaten by the user using the mobile device;calibrating a size of the food items in the image; segmenting the firstimage into one or more segmented regions where different food items arelocated; identifying the food of the segmented regions; estimating thevolume of food in the segmented regions of the first image.
 2. Themethod of claim 1, further comprising calculating the nutritionalinformation of the food items of the first image and communicating thisinformation to the user before the user starts eating.
 3. The method ofclaim 1, further comprising determining an amount of each food itemconsumed by the user by capturing a second image of any remaining fooditems after the user has eaten and repeating the calibrating,segmenting, identifying, and estimating steps for the food items in thesecond image, and then subtracting the remaining volume of each fooditem in the second image from the original volume of each food item inthe first image to determine the amount of each food item eaten.
 4. Themethod of claim 3, further comprising calculating nutritionalinformation based on the amount and identity of each food item eaten. 5.The method of claim 1, further comprising the selection by the user ofan option given by the mobile device for indicating that all food wasconsumed.
 6. The method of claim 1 where in the mobile device isnetwork-connected and allows for the transferring and receiving ofinformation.
 7. The method of claim 1 wherein the user labels theidentity of the food on at least the first image.
 8. The method of claim1 wherein the step of identifying the food comprises comparing the imageof the food taken by the user to images of known food items in a fooddatabase.
 9. The method of claim 1 wherein the image includes a fiducialmarker of a known size as a reference for size or volume calculations.10. The method of claim 1 for measuring beverage intake rather than foodintake.
 11. The method of claim 1 wherein the user can enter foodidentity and volume estimation manually into the mobile device the whenan image cannot be captured or is insufficient; and wherein thenutritional content of food items identified can be calculated usingknown measurements for food items in a food database.